Reliability and Comparability of Peer Review Results

In this paper peer review reliability is investigated based on peer ratings of research teams at two Belgian universities. It is found that outcomes can be substantially influenced by the different ways in which experts attribute ratings. To increase reliability of peer ratings, procedures creating a uniform reference level should be envisaged. One should at least check for signs of low reliability, which can be obtained from an analysis of the outcomes of the peer evaluation itself. The peer review results are compared to outcomes from a citation analysis of publications by the same teams, in subject fields well covered by citation indexes. It is illustrated how, besides reliability, comparability of results depends on the nature of the indicators, on the subject area and on the intrinsic characteristics of the methods. The results further confirm what is currently considered as good practice: the presentation of results for not one but for a series of indicators. While peer review is generally accepted as the principal method to evaluate research quality, its sensitivity to bias is well known. Several authors have investigated various kinds of bias in several review processes for grant and fellowship applications, manuscripts and the past performance of departments. Langfeldt (2004) e.g. distinguishes four categories of bias in peer review based on an in-depth analysis of six international evaluations of Norwegian research. Peers may not only have different or biased opinions, they may also convert their opinions into ratings in different ways. This is important in particular when not only quality labels are given, but also relative positions or rankings are derived from the experts' ratings. The influence of an expert's bias or rating habits can be limited by including several reviews for each case being reviewed. A further possibility is asking each expert to judge more than one file, creating a uniform reference level for the ratings and discouraging some kinds of bias by making them more visible. Both mechanisms were used in the evaluations discussed in this paper. It is not always feasible however to implement procedures avoiding or correcting for biased opinions or diverging rating habits, particularly when large numbers of cases are evaluated. It is therefore important to be able to examine whether results are sufficiently reliable and take additional steps whenever necessary. A discussion of studies on peer review reliability and on the combination of results from peer review and citation analysis is given …